3D pointing devices with orientation compensation and improved usability

ABSTRACT

Systems and methods according to the present invention describe 3D pointing devices which enhance usability by transforming sensed motion data from a first frame of reference (e.g., the body of the 3D pointing device) into a second frame of reference (e.g., a user&#39;s frame of reference). One exemplary embodiment of the present invention removes effects associated with a tilt orientation in which the 3D pointing device is held by a user.

RELATED APPLICATIONS

This application is a continuation of application Ser. No. 11/820,517,filed on Jun. 20, 2007, which is a continuation of application Ser. No.11/640,677, filed Dec. 18, 2006 and which issued on Aug. 28, 2007 asU.S. Pat. No. 7,262,760, which is a continuation of application Ser. No.11/119,719, filed May 2, 2005 and which issued on Jan. 2, 2007 as U.S.Pat. No. 7,158,118, which is related to, and claims priority from, U.S.Provisional Patent Application Ser. No. 60/566,444 filed on Apr. 30,2004, entitled “Freespace Pointing Device”, the disclosure of which isincorporated here by reference. This application is also related to, andclaims priority from, U.S. Provisional Patent Application Ser. No.60/612,571, filed on Sep. 23, 2004, entitled “Free Space PointingDevices and Methods”, the disclosure of which is incorporated here byreference. This application is also related to, and claims priorityfrom, U.S. Provisional Patent Application Ser. No. 60/641,410, filed onJan. 5, 2005, entitled “Freespace Pointing Devices and Methods for UsingSame”, the disclosure of which is incorporated here by reference. Thisapplication is also related to U.S. patent applications Ser. Nos.11/119,987, 11/119,688, and 11/119,663, entitled “Methods and Devicesfor Removing Unintentional Movement in 3D Pointing Devices”, “Methodsand Devices for Identifying Users Based on Tremor”, and “3D PointingDevices and Methods”, all of which were filed concurrently withapplication Ser. No. 11/119,719 on May 2, 2005, and all of which areincorporated here by reference.

BACKGROUND

The present invention relates generally to handheld, pointing devicesand, more specifically to three-dimensional (hereinafter “3D”) pointingdevices and techniques for tilt compensation and improved usabilityassociated therewith.

Technologies associated with the communication of information haveevolved rapidly over the last several decades. Television, cellulartelephony, the Internet and optical communication techniques (to namejust a few things) combine to inundate consumers with availableinformation and entertainment options. Taking television as an example,the last three decades have seen the introduction of cable televisionservice, satellite television service, pay-per-view movies andvideo-on-demand. Whereas television viewers of the 1960s could typicallyreceive perhaps four or five over-the-air TV channels on theirtelevision sets, today's TV watchers have the opportunity to select fromhundreds, thousands, and potentially millions of channels of shows andinformation. Video-on-demand technology, currently used primarily inhotels and the like, provides the potential for in-home entertainmentselection from among thousands of movie titles.

The technological ability to provide so much information and content toend users provides both opportunities and challenges to system designersand service providers. One challenge is that while end users typicallyprefer having more choices rather than fewer, this preference iscounterweighted by their desire that the selection process be both fastand simple. Unfortunately, the development of the systems and interfacesby which end users access media items has resulted in selectionprocesses which are neither fast nor simple. Consider again the exampleof television programs. When television was in its infancy, determiningwhich program to watch was a relatively simple process primarily due tothe small number of choices. One would consult a printed guide which wasformatted, for example, as series of columns and rows which showed thecorrespondence between (1) nearby television channels, (2) programsbeing transmitted on those channels and (3) date and time. Thetelevision was tuned to the desired channel by adjusting a tuner knoband the viewer watched the selected program. Later, remote controldevices were introduced that permitted viewers to tune the televisionfrom a distance. This addition to the user-television interface createdthe phenomenon known as “channel surfing” whereby a viewer could rapidlyview short segments being broadcast on a number of channels to quicklylearn what programs were available at any given time.

Despite the fact that the number of channels and amount of viewablecontent has dramatically increased, the generally available userinterface, control device options and frameworks for televisions has notchanged much over the last 30 years. Printed guides are still the mostprevalent mechanism for conveying programming information. The multiplebutton remote control with up and down arrows is still the mostprevalent channel/content selection mechanism. The reaction of those whodesign and implement the TV user interface to the increase in availablemedia content has been a straightforward extension of the existingselection procedures and interface objects. Thus, the number of rows inthe printed guides has been increased to accommodate more channels. Thenumber of buttons on the remote control devices has been increased tosupport additional functionality and content handling, e.g., as shown inFIG. 1. However, this approach has significantly increased both the timerequired for a viewer to review the available information and thecomplexity of actions required to implement a selection. Arguably, thecumbersome nature of the existing interface has hampered commercialimplementation of some services, e.g., video-on-demand, since consumersare resistant to new services that will add complexity to an interfacethat they view as already too slow and complex.

In addition to increases in bandwidth and content, the user interfacebottleneck problem is being exacerbated by the aggregation oftechnologies. Consumers are reacting positively to having the option ofbuying integrated systems rather than a number of segregable components.An example of this trend is the combination television/VCR/DVD in whichthree previously independent components are frequently sold today as anintegrated unit. This trend is likely to continue, potentially with anend result that most if not all of the communication devices currentlyfound in the household will be packaged together as an integrated unit,e.g., a television/VCR/DVD/internet access/radio/stereo unit. Even thosewho continue to buy separate components will likely desire seamlesscontrol of, and interworking between, the separate components. With thisincreased aggregation comes the potential for more complexity in theuser interface. For example, when so-called “universal” remote unitswere introduced, e.g., to combine the functionality of TV remote unitsand VCR remote units, the number of buttons on these universal remoteunits was typically more than the number of buttons on either the TVremote unit or VCR remote unit individually. This added number ofbuttons and functionality makes it very difficult to control anythingbut the simplest aspects of a TV or VCR without hunting for exactly theright button on the remote. Many times, these universal remotes do notprovide enough buttons to access many levels of control or featuresunique to certain TVs. In these cases, the original device remote unitis still needed, and the original hassle of handling multiple remotesremains due to user interface issues arising from the complexity ofaggregation. Some remote units have addressed this problem by adding“soft” buttons that can be programmed with the expert commands. Thesesoft buttons sometimes have accompanying LCD displays to indicate theiraction. These too have the flaw that they are difficult to use withoutlooking away from the TV to the remote control. Yet another flaw inthese remote units is the use of modes in an attempt to reduce thenumber of buttons. In these “moded” universal remote units, a specialbutton exists to select whether the remote should communicate with theTV, DVD player, cable set-top box, VCR, etc. This causes many usabilityissues including sending commands to the wrong device, forcing the userto look at the remote to make sure that it is in the right mode, and itdoes not provide any simplification to the integration of multipledevices. The most advanced of these universal remote units provide someintegration by allowing the user to program sequences of commands tomultiple devices into the remote. This is such a difficult task thatmany users hire professional installers to program their universalremote units.

Some attempts have also been made to modernize the screen interfacebetween end users and media systems. However, these attempts typicallysuffer from, among other drawbacks, an inability to easily scale betweenlarge collections of media items and small collections of media items.For example, interfaces which rely on lists of items may work well forsmall collections of media items, but are tedious to browse for largecollections of media items. Interfaces which rely on hierarchicalnavigation (e.g., tree structures) may be speedier to traverse than listinterfaces for large collections of media items, but are not readilyadaptable to small collections of media items. Additionally, users tendto lose interest in selection processes wherein the user has to movethrough three or more layers in a tree structure. For all of thesecases, current remote units make this selection processor even moretedious by forcing the user to repeatedly depress the up and downbuttons to navigate the list or hierarchies. When selection skippingcontrols are available such as page up and page down, the user usuallyhas to look at the remote to find these special buttons or be trained toknow that they even exist. Accordingly, organizing frameworks,techniques and systems which simplify the control and screen interfacebetween users and media systems as well as accelerate the selectionprocess, while at the same time permitting service providers to takeadvantage of the increases in available bandwidth to end user equipmentby facilitating the supply of a large number of media items and newservices to the user have been proposed in U.S. patent application Ser.No. 10/768,432, filed on Jan. 30, 2004, entitled “A Control Frameworkwith a Zoomable Graphical User Interface for Organizing, Selecting andLaunching Media Items”, the disclosure of which is incorporated here byreference.

Of particular interest for this specification are the remote devicesusable to interact with such frameworks, as well as other applicationsand systems. As mentioned in the above-incorporated application, variousdifferent types of remote devices can be used with such frameworksincluding, for example, trackballs, “mouse”-type pointing devices, lightpens, etc. However, another category of remote devices which can be usedwith such frameworks (and other applications) is 3D pointing devices.The phrase “3D pointing” is used in this specification to refer to theability of an input device to move in three (or more) dimensions in theair in front of, e.g., a display screen, and the corresponding abilityof the user interface to translate those motions directly into userinterface commands, e.g., movement of a cursor on the display screen.The transfer of data between the 3D pointing device may be performedwirelessly or via a wire connecting the 3D pointing device to anotherdevice. Thus “3D pointing” differs from, e.g., conventional computermouse pointing techniques which use a surface, e.g., a desk surface ormousepad, as a proxy surface from which relative movement of the mouseis translated into cursor movement on the computer display screen. Anexample of a 3D pointing device can be found in U.S. Pat. No. 5,440,326.

The '326 patent describes, among other things, a vertical gyroscopeadapted for use as a pointing device for controlling the position of acursor on the display of a computer. A motor at the core of thegyroscope is suspended by two pairs of orthogonal gimbals from ahand-held controller device and nominally oriented with its spin axisvertical by a pendulous device. Electro-optical shaft angle encoderssense the orientation of a hand-held controller device as it ismanipulated by a user and the resulting electrical output is convertedinto a format usable by a computer to control the movement of a cursoron the screen of the computer display.

However, the freedom of use associated with 3D pointers createsadditional challenges. For example, since there is generally no proxysurface on which a 3D pointing device rests, the orientation of thehandheld control device may vary considerably from user to user or evenuse to use. If a 3D pointing device is used to, for example, control themovement of a cursor displayed on a screen, then some mapping isperformed between the detected movement of the handheld device and themovement of the cursor on the screen.

One technique for performing this mapping is to use the body frame ofthe device as the frame of reference for mapping detected motion of the3D pointing device into intended motion of the cursor. The term “bodyframe” refers to a set of axes associated with the body of the objectbeing moved as described in more detail below. Using the body frame ofreference to perform the mapping, however, has certain drawbacks. Forexample, it requires the user to hold the device in a certainorientation in order to obtain the cursor movement he or she desires.For example, if the user holds the device on its side and moves thedevice left to right, the cursor will move vertically, not horizontally,on the screen.

Accordingly, the present invention describes methods and devices forprocessing the data received from sensor(s) in a manner which addressesthese and other problems associated with conventional 3D pointingdevices.

SUMMARY

Systems and methods according to the present invention describe 3Dpointing devices which enhance usability by transforming sensed motiondata from a first frame of reference (e.g., the body of the 3D pointingdevice) into a second frame of reference (e.g., a user's frame ofreference). One exemplary embodiment of the present invention removeseffects associated with a tilt orientation in which the 3D pointingdevice is held by a user.

According to an exemplary embodiment of the present invention, ahandheld, pointing device includes a first rotational sensor fordetermining rotation of the pointing device about a first axis andgenerating a first rotational output associated therewith, a secondrotational sensor for determining rotation of the pointing device abouta second axis and generating a second rotational output associatedtherewith, an accelerometer for determining an acceleration of thepointing device and outputting an acceleration output associatedtherewith and a processing unit for receiving the first and secondrotational outputs and the acceleration output and for: (a) convertingthe first and second rotational outputs and the acceleration output froma body frame of reference associated with the handheld pointing deviceinto a user's frame of reference in order to remove the effects of tiltassociated with the manner in which a user is holding the handheld,pointing device; and (b) determining data associated with x and ycoordinates which are in turn associated with movement of a screencursor, the data based on the converted first and second rotationaloutputs and the converted acceleration output, wherein the step ofconverting renders the movement of the screen cursor substantiallyindependent of an orientation in which a user holds the handheld device.

According to another exemplary embodiment of the present invention, amethod for using a 3D pointing device includes the steps of detectingmovement of the 3D pointing device and compensating the detectedmovement by transforming the detected movement from a body frame ofreference associated with the 3D pointing device into an inertial frameof reference.

According to yet another exemplary embodiment of the present invention,a 3D, handheld device includes at least one sensor for detectingmovement of the 3D pointing device and a processing unit forcompensating the detected movement by transforming the detected movementfrom a body frame of reference associated with the 3D pointing deviceinto an inertial frame of reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate exemplary embodiments of thepresent invention, wherein:

FIG. 1 depicts a conventional remote control unit for an entertainmentsystem;

FIG. 2 depicts an exemplary media system in which exemplary embodimentsof the present invention can be implemented;

FIG. 3 shows a 3D pointing device according to an exemplary embodimentof the present invention;

FIG. 4 illustrates a cutaway view of the 3D pointing device in FIG. 4including two rotational sensors and one accelerometer;

FIG. 5 is a block diagram illustrating processing of data associatedwith 3D pointing devices according to an exemplary embodiment of thepresent invention;

FIGS. 6( a)-6(d) illustrate the effects of tilt;

FIG. 7 depicts a hardware architecture of a 3D pointing device accordingto an exemplary embodiment of the present invention;

FIG. 8 is a state diagram depicting a stationary detection mechanismaccording to an exemplary embodiment of the present invention;

FIG. 9 is a block diagram illustrating transformation of sensed motiondata from a first frame of reference into a second frame of referenceaccording to an exemplary embodiment of the present invention; and

FIG. 10 graphically illustrates the transformation of sensed motion datafrom a first frame of reference into a second frame of referenceaccording to an exemplary embodiment of the present invention.

DETAILED DESCRIPTION

The following detailed description of the invention refers to theaccompanying drawings. The same reference numbers in different drawingsidentify the same or similar elements. Also, the following detaileddescription does not limit the invention. Instead, the scope of theinvention is defined by the appended claims.

In order to provide some context for this discussion, an exemplaryaggregated media system 200 in which the present invention can beimplemented will first be described with respect to FIG. 2. Thoseskilled in the art will appreciate, however, that the present inventionis not restricted to implementation in this type of media system andthat more or fewer components can be included therein. Therein, aninput/output (I/O) bus 210 connects the system components in the mediasystem 200 together. The I/O bus 210 represents any of a number ofdifferent of mechanisms and techniques for routing signals between themedia system components. For example, the I/O bus 210 may include anappropriate number of independent audio “patch” cables that route audiosignals, coaxial cables that route video signals, two-wire serial linesor infrared or radio frequency transceivers that route control signals,optical fiber or any other routing mechanisms that route other types ofsignals.

In this exemplary embodiment, the media system 200 includes atelevision/monitor 212, a video cassette recorder (VCR) 214, digitalvideo disk (DVD) recorder/playback device 216, audio/video tuner 218 andcompact disk player 220 coupled to the I/O bus 210. The VCR 214, DVD 216and compact disk player 220 may be single disk or single cassettedevices, or alternatively may be multiple disk or multiple cassettedevices. They may be independent units or integrated together. Inaddition, the media system 200 includes a microphone/speaker system 222,video camera 224 and a wireless I/O control device 226. According toexemplary embodiments of the present invention, the wireless I/O controldevice 226 is a 3D pointing device according to one of the exemplaryembodiments described below. The wireless I/O control device 226 cancommunicate with the entertainment system 200 using, e.g., an IR or RFtransmitter or transceiver. Alternatively, the I/O control device can beconnected to the entertainment system 200 via a wire.

The entertainment system 200 also includes a system controller 228.According to one exemplary embodiment of the present invention, thesystem controller 228 operates to store and display entertainment systemdata available from a plurality of entertainment system data sources andto control a wide variety of features associated with each of the systemcomponents. As shown in FIG. 2, system controller 228 is coupled, eitherdirectly or indirectly, to each of the system components, as necessary,through I/O bus 210. In one exemplary embodiment, in addition to or inplace of I/O bus 210, system controller 228 is configured with awireless communication transmitter (or transceiver), which is capable ofcommunicating with the system components via IR signals or RF signals.Regardless of the control medium, the system controller 228 isconfigured to control the media components of the media system 200 via agraphical user interface described below.

As further illustrated in FIG. 2, media system 200 may be configured toreceive media items from various media sources and service providers. Inthis exemplary embodiment, media system 200 receives media input fromand, optionally, sends information to, any or all of the followingsources: cable broadcast 230, satellite broadcast 232 (e.g., via asatellite dish), very high frequency (VHF) or ultra high frequency (UHF)radio frequency communication of the broadcast television networks 234(e.g., via an aerial antenna), telephone network 236 and cable modem 238(or another source of Internet content). Those skilled in the art willappreciate that the media components and media sources illustrated anddescribed with respect to FIG. 2 are purely exemplary and that mediasystem 200 may include more or fewer of both. For example, other typesof inputs to the system include AM/FM radio and satellite radio.

More details regarding this exemplary entertainment system andframeworks associated therewith can be found in the above-incorporatedby reference U.S. Patent Application “A Control Framework with aZoomable Graphical User Interface for Organizing, Selecting andLaunching Media Items”. Alternatively, remote devices in accordance withthe present invention can be used in conjunction with other systems, forexample computer systems including, e.g., a display, a processor and amemory system or with various other systems and applications.

As mentioned in the Background section, remote devices which operate as3D pointers are of particular interest for the present specification.Such devices enable the translation of movement, e.g., gestures, intocommands to a user interface. An exemplary 3D pointing device 400 isdepicted in FIG. 3. Therein, user movement of the 3D pointing can bedefined, for example, in terms of a combination of x-axis attitude(roll), y-axis elevation (pitch) and/or z-axis heading (yaw) motion ofthe 3D pointing device 400. In addition, some exemplary embodiments ofthe present invention can also measure linear movement of the 3Dpointing device 400 along the x, y, and z axes to generate cursormovement or other user interface commands. In the exemplary embodimentof FIG. 3, the 3D pointing device 400 includes two buttons 402 and 404as well as a scroll wheel 406, although other exemplary embodiments willinclude other physical configurations. According to exemplaryembodiments of the present invention, it is anticipated that 3D pointingdevices 400 will be held by a user in front of a display 408 and thatmotion of the 3D pointing device 400 will be translated by the 3Dpointing device into output which is usable to interact with theinformation displayed on display 408, e.g., to move the cursor 410 onthe display 408. For example, rotation of the 3D pointing device 400about the y-axis can be sensed by the 3D pointing device 400 andtranslated into an output usable by the system to move cursor 410 alongthe y₂ axis of the display 408. Likewise, rotation of the 3D pointingdevice 408 about the z-axis can be sensed by the 3D pointing device 400and translated into an output usable by the system to move cursor 410along the x₂ axis of the display 408. It will be appreciated that theoutput of 3D pointing device 400 can be used to interact with thedisplay 408 in a number of ways other than (or in addition to) cursormovement, for example it can control cursor fading, volume or mediatransport (play, pause, fast-forward and rewind). Input commands mayinclude operations in addition to cursor movement, for example, a zoomin or zoom out on a particular region of a display. A cursor may or maynot be visible. Similarly, rotation of the 3D pointing device 400 sensedabout the x-axis of 3D pointing device 400 can be used in addition to,or as an alternative to, y-axis and/or z-axis rotation to provide inputto a user interface.

According to one exemplary embodiment of the present invention, tworotational sensors 502 and 504 and one accelerometer 506 can be employedas sensors in 3D pointing device 400 as shown in FIG. 4. The rotationalsensors 502 and 504 can, for example, be implemented using ADXRS150 orADXRS401 sensors made by Analog Devices. It will be appreciated by thoseskilled in the art that other types of rotational sensors can beemployed as rotational sensors 502 and 504 and that the ADXRS150 andADXRS401 are purely used as an illustrative example. Unlike traditionalgyroscopes, the ADXRS150 rotational sensors use MEMS technology toprovide a resonating mass which is attached to a frame so that it canresonate only along one direction. The resonating mass is displaced whenthe body to which the sensor is affixed is rotated around the sensor'ssensing axis. This displacement can be measured using the Coriolisacceleration effect to determine an angular velocity associated withrotation along the sensing axis. If the rotational sensors 502 and 504have a single sensing axis (as for example the ADXRS150s), then they canbe mounted in the 3D pointing device 400 such that their sensing axesare aligned with the rotations to be measured. For this exemplaryembodiment of the present invention, this means that rotational sensor504 is mounted such that its sensing axis is parallel to the y-axis andthat rotational sensor 502 is mounted such that its sensing axis isparallel to the z-axis as shown in FIG. 4. Note, however, that aligningthe sensing axes of the rotational sensors 502 and 504 parallel to thedesired measurement axes is not required since exemplary embodiments ofthe present invention also provide techniques for compensating foroffset between axes.

One challenge faced in implementing exemplary 3D pointing devices 400 inaccordance with the present invention is to employ components, e.g.,rotational sensors 502 and 504, which are not too costly, while at thesame time providing a high degree of correlation between movement of the3D pointing device 400, a user's expectation regarding how the userinterface will react to that particular movement of the 3D pointingdevice and actual user interface performance in response to thatmovement. For example, if the 3D pointing device 400 is not moving, theuser will likely expect that the cursor ought not to be drifting acrossthe screen. Likewise, if the user rotates the 3D pointing device 400purely around the y-axis, she or he would likely not expect to see theresulting cursor movement on display 408 contain any significant x₂ axiscomponent. To achieve these, and other, aspects of exemplary embodimentsof the present invention, various measurements and calculations areperformed by the handheld device 400 which are used to adjust theoutputs of one or more of the sensors 502, 504 and 506 and/or as part ofthe input used by a processor to determine an appropriate output for theuser interface based on the outputs of the sensors 502, 504 and 506.These measurements and calculations are used to compensate for factorswhich fall broadly into two categories: (1) factors which are intrinsicto the 3D pointing device 400, e.g., errors associated with theparticular sensors 502, 504 and 506 used in the device 400 or the way inwhich the sensors are mounted in the device 400 and (2) factors whichare not intrinsic to the 3D pointing device 400, but are insteadassociated with the manner in which a user is using the 3D pointingdevice 400, e.g., linear acceleration, tilt and tremor. Exemplarytechniques for handling each of these effects are described below.

A process model 600 which describes the general operation of 3D pointingdevices according to exemplary embodiments of the present invention isillustrated in FIG. 5. The rotational sensors 502 and 504, as well asthe accelerometer 506, produce analog signals which are sampledperiodically, e.g., 200 samples/second. For the purposes of thisdiscussion, a set of these inputs shall be referred to using thenotation (x, y, z, αy, αz), wherein x, y, z are the sampled outputvalues of the exemplary three-axis accelerometer 506 which areassociated with acceleration of the 3D pointing device in the x-axis,y-axis and z-axis directions, respectively, αy is a the sampled outputvalue from rotational sensor 502 associated with the rotation of the 3Dpointing device about the y-axis and αz is the sampled output value fromrotational sensor 504 associated with rotation of the 3D pointing device400 about the z-axis.

The output from the accelerometer 506 is provided and, if theaccelerometer 506 provides analog output, then the output is sampled anddigitized by an A/D converter (not shown) to generate sampledaccelerometer output 602. The sampled output values are converted fromraw units to units of acceleration, e.g., gravities (g), as indicated byconversion function 604. The acceleration calibration block 606 providesthe values used for the conversion function 604. This calibration of theaccelerometer output 602 can include, for example, compensation for oneor more of scale, offset and axis misalignment error associated with theaccelerometer 506. Exemplary conversions for the accelerometer data canbe performed using the following equation:A=S*((M−P)·*G(T))  (1)wherein M is a 3×1 column vector composed of the sampled output values(x, y, z), P is a 3×1 column vector of sensor offsets, and S is a 3×3matrix that contains both scale, axis misalignment, and sensor rotationcompensation. G(T) is a gain factor that is a function of temperature.The “*” operator represents matrix multiplication and the “·*” operatorrepresents element multiplication. The exemplary accelerometer 506 hasan exemplary full range of +/−2 g. Sensor offset, P, refers to thesensor output, M, for an accelerometer measurement of 0 g. Scale refersto the conversion factor between the sampled unit value and g. Theactual scale of any given accelerometer sensor may deviate from thesenominal scale values due to, e.g., manufacturing variances. Accordinglythe scale factor in the equations above will be proportional to thisdeviation.

Accelerometer 506 scale and offset deviations can be measured by, forexample, applying 1 g of force along one an axis and measuring theresult, R1. Then a—1 g force is applied resulting in measurement R2. Theindividual axis scale, s, and the individual axis offset, p, can becomputed as follows:s=(R1−R2)/2  (2)p=(R1+R2)/2  (3)In this simple case, P is the column vector of the p for each axis, andS is the diagonal matrix of the 1/s for each axis.

However, in addition to scale and offset, readings generated byaccelerometer 506 may also suffer from cross-axes effects. Cross-axeseffects include non-aligned axes, e.g., wherein one or more of thesensing axes of the accelerometer 506 as it is mounted in the 3Dpointing device 400 are not aligned with the corresponding axis in theinertial frame of reference, or mechanical errors associated with themachining of the accelerometer 506 itself, e.g., wherein even though theaxes are properly aligned, a purely y-axis acceleration force may resultin a sensor reading along the z-axis of the accelerometer 506. Both ofthese effects can also be measured and added to the calibrationperformed by function 606.

The accelerometer 506 serves several purposes in exemplary 3D pointingdevices according to exemplary embodiments of the present invention. Forexample, if rotational sensors 502 and 504 are implemented using theexemplary Coriolis effect rotational sensors described above, then theoutput of the rotational sensors 502 and 504 will vary based on thelinear acceleration experienced by each rotational sensor. Thus, oneexemplary use of the accelerometer 506 is to compensate for fluctuationsin the readings generated by the rotational sensors 502 and 504 whichare caused by variances in linear acceleration. This can be accomplishedby multiplying the converted accelerometer readings by a gain matrix 610and subtracting (or adding) the results from (or to) the correspondingsampled rotational sensor data 612. For example, the sampled rotationaldata αy from rotational sensor 502 can be compensated for linearacceleration at block 614 as:αy′=αy−C*A  (4)wherein C is the 1×3 row vector of rotational sensor susceptibility tolinear acceleration along each axis given in units/g and A is thecalibrated linear acceleration. Similarly, linear accelerationcompensation for the sampled rotational data αz from rotational sensor504 can be provided at block 614. The gain matrices, C, vary betweenrotational sensors due to manufacturing differences. C may be computedusing the average value for many rotational sensors, or it may be customcomputed for each rotational sensor.

Like the accelerometer data, the sampled rotational data 612 is thenconverted from a sampled unit value into a value associated with a rateof angular rotation, e.g., radians/s, at function 616. This conversionstep can also include calibration provided by function 618 to compensatethe sampled rotational data for, e.g., scale and offset.Conversion/calibration for both αy and αz can be accomplished using, forexample, the following equation:αrad/s=(α′−offset(T))*scale+dOffset  (5)wherein α′ refers to the value being converted/calibrated, offset(T)refers to an offset value associated with temperature, scale refers tothe conversion factor between the sampled unit value and rad/s, anddOffset refers to a dynamic offset value. Equation (5) may beimplemented as a matrix equation in which case all variables are vectorsexcept for scale. In matrix equation form, scale corrects for axismisalignment and rotational offset factors. Each of these variables isdiscussed in more detail below.

The offset values offset(T) and dOffset can be determined in a number ofdifferent ways. When the 3D pointing device 400 is not being rotated in,for example, the y-axis direction, the sensor 502 should output itsoffset value. However, the offset can be highly affected by temperature,so this offset value will likely vary. Offset temperature calibrationmay be performed at the factory, in which case the value(s) foroffset(T) can be preprogrammed into the handheld device 400 or,alternatively, offset temperature calibration may also be learneddynamically during the lifetime of the device. To accomplish dynamicoffset compensation, an input from a temperature sensor 619 is used inrotation calibration function 618 to compute the current value foroffset(T). The offset(T) parameter removes the majority of offset biasfrom the sensor readings. However, negating nearly all cursor drift atzero movement can be useful for producing a high-performance pointingdevice. Therefore, the additional factor dOffset, can be computeddynamically while the 3D pointing device 400 is in use. The stationarydetection function 608 determines when the handheld is most likelystationary and when the offset should be recomputed. Exemplarytechniques for implementing stationary detection function 608, as wellas other uses therefore, are described below.

An exemplary implementation of dOffset computation employs calibratedsensor outputs which are low-pass filtered. The stationary outputdetection function 608 provides an indication to rotation calibrationfunction 618 to trigger computation of, for example, the mean of thelow-pass filter output. The stationary output detection function 608 canalso control when the newly computed mean is factored into the existingvalue for dOffset. Those skilled in the art will recognize that amultitude of different techniques can be used for computing the newvalue for dOffset from the existing value of dOffset and the new meanincluding, but not limited to, simple averaging, low-pass filtering andKalman filtering. Additionally, those skilled in the art will recognizethat numerous variations for offset compensation of the rotationalsensors 502 and 504 can be employed. For example, the offset(T) functioncan have a constant value (e.g., invariant with temperature), more thantwo offset compensation values can be used and/or only a single offsetvalue can be computed/used for offset compensation.

After conversion/calibration at block 616, the inputs from therotational sensors 502 and 504 can be further processed to rotate thoseinputs into an inertial frame of reference, i.e., to compensate for tiltassociated with the manner in which the user is holding the 3D pointingdevice 400, at function 620. Tilt correction is another significantaspect of some exemplary embodiments of the present invention as it isintended to compensate for differences in usage patterns of 3D pointingdevices according to the present invention. More specifically, tiltcorrection according to exemplary embodiments of the present inventionis intended to compensate for the fact that users will hold pointingdevices in their hands at different x-axis rotational positions, butthat the sensing axes of the rotational sensors 502 and 504 in the 3Dpointing devices 400 are fixed. It is desirable that cursor translationacross display 408 is substantially insensitive to the way in which theuser grips the 3D pointing device 400, e.g., rotating the 3D pointingdevice 400 back and forth in a manner generally corresponding to thehorizontal dimension (x₂-axis) of the display 508 should result incursor translation along the x₂-axis, while rotating the 3D pointingdevice up and down in a manner generally corresponding to the verticaldimension (y₂-axis) of the display 508 should result in cursortranslation along the y₂-axis, regardless of the orientation in whichthe user is holding the 3D pointing device 400.

To better understand the need for tilt compensation according toexemplary embodiments of the present invention, consider the exampleshown in FIG. 6( a). Therein, the user is holding 3D pointing device 400in an exemplary inertial frame of reference, which can be defined ashaving an x-axis rotational value of 0 degrees, e.g., the inertial frameof reference can that in which the 3D device has its bottomsubstantially parallel to a floor of a room in which, e.g., a televisionis located. The inertial frame of reference can, purely as an example,correspond to the orientation illustrated in FIG. 6( a) or it can bedefined as any other orientation. Rotation of the 3D pointing device 400in either the y-axis or z-axis directions will be sensed by rotationalsensors 502 and 504, respectively. For example, rotation of the 3Dpointing device 400 around the z-axis by an amount Δz as shown in FIG.6( b) will result in a corresponding cursor translation Δx₂ in the x₂axis dimension across the display 408 (i.e., the distance between thedotted version of cursor 410 and the undotted version).

If, on the other hand, the user holds the 3D pointing device 400 in adifferent orientation, e.g., with some amount of x-axis rotationrelative to the inertial frame of reference, then the informationprovided by the sensors 502 and 504 would not (absent tilt compensation)provide an accurate representation of the user's intended interfaceactions. For example, referring to FIG. 6( c), consider a situationwherein the user holds the 3D pointing device 400 with an x-axisrotation of 45 degrees relative to the exemplary inertial frame ofreference as illustrated in FIG. 6( a). Assuming the same z-axisrotation Az is imparted to the 3D pointing device 400 by a user as inthe example of FIG. 6( b), the cursor 410 will instead be translated inboth the x₂-axis direction and the y₂-axis direction as shown in FIG. 6(d). This is due to the fact that the sensing axis of rotational sensor502 is now oriented between the y-axis and the z-axis (because of theorientation of the device in the user's hand). Similarly, the sensingaxis of the rotational sensor 504 is also oriented between the y-axisand the z-axis (although in a different quadrant). In order to providean interface which is transparent to the user in terms of how the 3Dpointing device 400 is held, tilt compensation according to exemplaryembodiments of the present invention translates the readings output fromrotational sensors 502 and 504 back into the inertial frame of referenceas part of processing the readings from these sensors into informationindicative of rotational motion of the 3D pointing device 400.

According to exemplary embodiments of the present invention, returningto FIG. 5, this can be accomplished by determining the tilt of the 3Dpointing device 400 using the inputs y and z received from accelerometer506 at function 622. More specifically, after the acceleration data isconverted and calibrated as described above, it can be low pass filteredat LPF 624 to provide an average acceleration (gravity) value to thetilt determination function 622. Then, tilt θ can be calculated infunction 622 as:

$\begin{matrix}{\theta = {\tan^{- 1}( \frac{y}{z} )}} & (7)\end{matrix}$The value θ can be numerically computed as a tan 2(y,z) to preventdivision by zero and give the correct sign. Then, function 620 canperform the rotation R of the converted/calibrated inputs αy and αzusing the equation:

$\begin{matrix}{R = {\begin{bmatrix}{\cos\;\theta\;\sin\;\theta} \\{{- \sin}\;\theta\;\cos\;\theta}\end{bmatrix} \cdot \begin{bmatrix}{\alpha\; y} \\{\alpha\; z}\end{bmatrix}}} & (8)\end{matrix}$to rotate the converted/calibrated inputs αy and αz to compensate forthe tilt θ. Tilt compensation as described in this exemplary embodimentis a subset of a more general technique for translating sensor readingsfrom the body frame of reference into a user's frame of referenceaccording to another exemplary embodiment of the present invention whichis described below.

Once the calibrated sensor readings have been compensated for linearacceleration, processed into readings indicative of angular rotation ofthe 3D pointing device 400, and compensated for tilt, post-processingcan be performed at blocks 626 and 628. Exemplary post-processing caninclude compensation for various factors such as human tremor. Althoughtremor may be removed using several different methods, one way to removetremor is by using hysteresis. The angular velocity produced by rotationfunction 620 is integrated to produce an angular position. Hysteresis ofa calibrated magnitude is then applied to the angular position. Thederivative is taken of the output of the hysteresis block to again yieldan angular velocity. The resulting output is then scaled at function 628(e.g., based on the sampling period) and used to generate a resultwithin the interface, e.g., movement of a cursor 410 on a display 408.

Having provided a process description of exemplary 3D pointing devicesaccording to the present invention, FIG. 7 illustrates an exemplaryhardware architecture. Therein, a processor 800 communicates with otherelements of the 3D pointing device including a scroll wheel 802, JTAG804, LEDs 806, switch matrix 808, IR photodetector 810, rotationalsensors 812, accelerometer 814 and transceiver 816. The scroll wheel 802is an optional input component which enables a user to provide input tothe interface by rotating the scroll wheel 802 clockwise orcounterclockwise. JTAG 804 provides the programming and debugginginterface to the processor. LEDs 806 provide visual feedback to a user,for example, when a button is pressed. Switch matrix 808 receivesinputs, e.g., indications that a button on the 3D pointing device 400has been depressed or released, that are then passed on to processor800. The optional IR photodetector 810 can be provided to enable theexemplary 3D pointing device to learn IR codes from other remotecontrols. Rotational sensors 812 provide readings to processor 800regarding, e.g., the y-axis and z-axis rotation of the 3D pointingdevice as described above. Accelerometer 814 provides readings toprocessor 800 regarding the linear acceleration of the 3D pointingdevice 400 which can be used as described above, e.g., to perform tiltcompensation and to compensate for errors which linear accelerationintroduces into the rotational readings generated by rotational sensors812. Transceiver 816 is used to communicate information to and from 3Dpointing device 400, e.g., to the system controller 228 or to aprocessor associated with a computer. The transceiver 816 can be awireless transceiver, e.g., operating in accordance with the Bluetoothstandards for short-range wireless communication or an infraredtransceiver. Alternatively, 3D pointing device 400 can communicate withsystems via a wireline connection.

In the exemplary embodiment of FIG. 4, the 3D pointing device 400includes two rotational sensors 502 and 504, as well as an accelerometer506. However, according to another exemplary embodiment of the presentinvention, a 3D pointing device can alternatively include just onerotational sensor, e.g., for measuring angular velocity in the z-axisdirection, and an accelerometer. For such an exemplary embodiment,similar functionality to that described above can be provided by usingthe accelerometer to determine the angular velocity along the axis whichis not sensed by the rotational sensor. For example, rotational velocityaround the y-axis can be computed using data generated by theaccelerometer and calculating:

$\begin{matrix}{\omega_{Y} = {\frac{\partial\theta_{Y}}{\partial t} = {\frac{\partial}{\partial t}{\tan^{- 1}( \frac{x}{z} )}}}} & (9)\end{matrix}$In addition, the parasitic acceleration effects that are not measured bya rotational sensor should also be removed. These effects include actuallinear acceleration, acceleration measured due to rotational velocityand rotational acceleration, and acceleration due to human tremor.

Stationary detection function 608, mentioned briefly above, can operateto determine whether the 3D pointing device 400 is, for example, eitherstationary or active (moving). This categorization can be performed in anumber of different ways. One way, according to an exemplary embodimentof the present invention, is to compute the variance of the sampledinput data of all inputs (x, y, z, αy, αz) over a predetermined window,e.g., every quarter of a second. This variance is then compared with athreshold to classify the 3D pointing device as either stationary oractive.

Another stationary detection technique according to exemplaryembodiments of the present invention involves transforming the inputsinto the frequency domain by, e.g., performing a Fast Fourier Transform(FFT) on the input data. Then, the data can be analyzed using, e.g.,peak detection methods, to determine if the 3D pointing device 400 iseither stationary or active. Additionally, a third category can bedistinguished, specifically the case where a user is holding the 3Dpointing device 400 but is not moving it (also referred to herein as the“stable” state. This third category can be distinguished from stationary(not held) and active by detecting the small movement of the 3D pointingdevice 400 introduced by a user's hand tremor when the 3D pointingdevice 400 is being held by a user. Peak detection can also be used bystationary detection function 608 to make this determination. Peakswithin the range of human tremor frequencies, e.g., nominally 8-12 Hz,will typically exceed the noise floor of the device (experienced whenthe device is stationary and not held) by approximately 20 dB.

In the foregoing examples, the variances in the frequency domain weresensed within a particular frequency range, however the actual frequencyrange to be monitored and used to characterize the status of the 3Dpointing device 400 may vary. For example, the nominal tremor frequencyrange may shift based on e.g., the ergonomics and weight of the 3Dpointing device 400, e.g., from 8-12 Hz to 4-7 Hz.

According to another exemplary embodiment of the present invention,stationary detection mechanism 608 can include a state machine. Anexemplary state machine is shown in FIG. 8. Therein, the ACTIVE stateis, in this example, the default state during which the 3D pointingdevice 400 is moving and being used to, e.g., provide inputs to a userinterface. The 3D pointing device 400 can enter the ACTIVE state onpower-up of the device as indicated by the reset input. If the 3Dpointing device 400 stops moving, it may then enter the INACTIVE state.The various state transitions illustrated in FIG. 8 can be triggered byany of a number of different criteria including, but not limited to,data output from one or both of the rotational sensors 502 and 504, dataoutput from the accelerometer 506, time domain data, frequency domaindata or any combination thereof. State transition conditions will begenerically referred to herein using the convention “Condition_(stateA)_(→) _(stateB)”. For example, the 3D pointing device 400 will transitionfrom the ACTIVE state to the INACTIVE state when condition_(active) _(→)_(inactive) occurs. For the sole purpose of illustration, consider thatcondition_(active) _(→) _(inactive) can, in an exemplary 3D pointingdevice 400, occur when mean and/or standard deviation values from boththe rotational sensor(s) and the accelerometer fall below firstpredetermined threshold values for a first predetermined time period.

State transitions can be determined by a number of different conditionsbased upon the interpreted sensor outputs. Exemplary condition metricsinclude the variance of the interpreted signals over a time window, thethreshold between a reference value and the interpreted signal over atime window, the threshold between a reference value and the filteredinterpreted signal over a time window, and the threshold between areference value and the interpreted signal from a start time can be usedto determine state transitions. All, or any combination, of thesecondition metrics can be used to trigger state transitions.Alternatively, other metrics can also be used. According to oneexemplary embodiment of the present invention, a transition from theINACTIVE state to the ACTIVE state occurs either when (1) a mean valueof sensor output(s) over a time window is greater than predeterminedthreshold(s) or (2) a variance of values of sensor output(s) over a timewindow is greater than predetermined threshold(s) or (3) aninstantaneous delta between sensor values is greater than apredetermined threshold.

The INACTIVE state enables the stationary detection mechanism 608 todistinguish between brief pauses during which the 3D pointing device 400is still being used, e.g., on the order of a tenth of a second, and anactual transition to either a stable or stationary condition. Thisprotects against the functions which are performed during the STABLE andSTATIONARY states, described below, from inadvertently being performedwhen the 3D pointing device is being used. The 3D pointing device 400will transition back to the ACTIVE state when condition_(inactive) _(→)_(active) occurs, e.g., if the 3D pointing device 400 starts movingagain such that the measured outputs from the rotational sensor(s) andthe accelerometer exceeds the first threshold before a secondpredetermined time period in the INACTIVE state elapses.

The 3D pointing device 400 will transition to either the STABLE state orthe STATIONARY state after the second predetermined time period elapses.As mentioned earlier, the STABLE state reflects the characterization ofthe 3D pointing device 400 as being held by a person but beingsubstantially unmoving, while the STATIONARY state reflects acharacterization of the 3D pointing device as not being held by aperson. Thus, an exemplary state machine according to the presentinvention can provide for a transition to the STABLE state after thesecond predetermined time period has elapsed if minimal movementassociated with hand tremor is present or, otherwise, transition to theSTATIONARY state.

The STABLE and STATIONARY states define times during which the 3Dpointing device 400 can perform various functions. For example, sincethe STABLE state is intended to reflect times when the user is holdingthe 3D pointing device 400 but is not moving it, the device can recordthe movement of the 3D pointing device 400 when it is in the STABLEstate e.g., by storing outputs from the rotational sensor(s) and/or theaccelerometer while in this state. These stored measurements can be usedto determine a tremor pattern associated with a particular user or usersas described below. Likewise, when in the STATIONARY state, the 3Dpointing device 400 can take readings from the rotational sensors and/orthe accelerometer for use in compensating for offset as described above.

If the 3D pointing device 400 starts to move while in either the STABLEor STATIONARY state, this can trigger a return to the ACTIVE state.Otherwise, after measurements are taken, the device can transition tothe SLEEP state. While in the sleep state, the device can enter a powerdown mode wherein power consumption of the 3D pointing device is reducedand, e.g., the sampling rate of the rotational sensors and/or theaccelerometer is also reduced. The SLEEP state can also be entered viaan external command so that the user or another device can command the3D pointing device 400 to enter the SLEEP state.

Upon receipt of another command, or if the 3D pointing device 400 beginsto move, the device can transition from the SLEEP state to the WAKEUPstate. Like the INACTIVE state, the WAKEUP state provides an opportunityfor the device to confirm that a transition to the ACTIVE state isjustified, e.g., that the 3D pointing device 400 was not inadvertentlyjostled.

The conditions for state transitions may be symmetrical or may differ.Thus, the threshold associated with the condition_(active) _(→)_(inactive) may be the same as (or different from) the threshold(s)associated with the condition_(inactive) _(→) _(active). This enables 3Dpointing devices according to the present invention to more accuratelycapture user input. For example, exemplary embodiments which include astate machine implementation allow, among other things, for thethreshold for transition into a stationary condition to be differentthan the threshold for the transition out of a stationary condition.

Entering or leaving a state can be used to trigger other devicefunctions as well. For example, the user interface can be powered upbased on a transition from any state to the ACTIVE state. Conversely,the 3D pointing device and/or the user interface can be turned off (orenter a sleep mode) when the 3D pointing device transitions from ACTIVEor STABLE to STATIONARY or INACTIVE. Alternatively, the cursor 410 canbe displayed or removed from the screen based on the transition from orto the stationary state of the 3D pointing device 400.

As mentioned above, exemplary embodiments of the present inventionprocess movement data received from sensor(s) in the 3D pointing deviceto convert this data from the frame of reference of the 3D pointingdevice's body into another frame of reference, e.g., the user's frame ofreference. In the exemplary application of a 3D pointing device used tocontrol a user interface displayed on a screen, e.g., a television, theuser's frame of reference might be a coordinate system associated withthe television screen. Regardless, translation of the data from the bodyframe of reference into another frame of reference improves theusability of the handheld device by resulting in an operation that isfrom the user's perspective rather than the device's perspective. Thus,when the user moves his or her hand from left to right in front of adisplay while holding the 3D pointing device, the cursor will move inthe left to right direction regardless of the orientation of the 3Dpointing device.

To simplify this discussion an exemplary processing system associatedwith a 3D pointing device is shown in FIG. 9, e.g., as described in moredetail above. Therein, the handheld system senses motion using one ormore sensors 901, e.g., rotational sensor(s), gyroscopes(s),accelerometer(s), magnetometer(s), optical sensor(s), camera(s) or anycombination thereof. The sensors are then interpreted in block 902 toproduce an estimate of the motion that occurred. The processing block903 then translates the measured motion from the natural (body)reference frame of the device into the reference frame of the user. Themovement is then mapped 904 into meaningful actions that are interpretedat block 905 forwarded to the system to produce a meaningful response,such as moving an on-screen cursor.

Block 903 converts detected movement into the reference frame of theuser instead of the reference frame of the device. Orientation may berepresented by many different mathematically similar methods includingEuler angles, a direction cosine matrix (DCM), or a unit quaternion.Position is generally represented as an offset from the coordinatesystem origin in a consistent unit including but not limited to meters,centimeters, feet, inches, and miles. In one exemplary embodimentdescribed above, a 3D pointing device measures inertial forces includingacceleration and rotational velocity. These forces are measured relativeto the body of the device by sensors mounted therein. In order toconvert the measured data into the user frame of reference, the deviceestimates both its position and its orientation.

In this exemplary embodiment, it is assumed that the user frame ofreference is stationary and has fixed orientation, although thoseskilled in the art will appreciate that this technique in accordancewith the present invention can be readily extended to the cases wherethe user's frame of reference is non-stationary by either directlytransforming to the time-varying frame or by first converting to astationary frame and then converting to the moving frame. For thestationary, fixed-orientation user frame of reference example,conversion from the body frame to the user frame can be performed by useof the following equations:Pu=Rotate(Pb,Q)+PdeltaPu′=Rotate(Pb′,Q)Pu″=Rotate(Pb″,Q)Wu=Rotate(Wb,Q)Wu′=Rotate(Wb′,Q)where:

Rotate represents the quaternion rotation operator such that Rotate(A,Q) is equal to Q* A Q where Q* is the quaternion conjugate and thevector A is a quaternion with the complex component equal to A and thereal component equal to 0;

Pu is the position in the user frame of reference;

Pb is the position in the device frame of reference;

′ represents the derivative. Therefore, Pu′ is the derivative of theposition in the user frame of reference which is the velocity in theuser frame of reference;

Wu is the angular velocity of the device in body angles in the userframe of reference;

Wb is the angular velocity of the device in body angles in the bodyframe of the device;

Pdelta is the difference between the origin of the user frame ofreference and the body frame of reference in the user frame of referencecoordinate system;

Q is the normalized rotation quaternion that represents the rotationfrom the body frame to the user frame. Since the rotation quaternion torotate from the user frame to the body frame is Q*, we could replace Qwith R* where R is the rotation from the user frame to the body frame.Note that Q can be represented in a number of equivalent forms includingEuler angles and the direction cosine matrix (DCM), and the aboveequations may vary slightly in their equivalent forms based upondifferent representations of Q. FIG. 10 graphically illustrates thetransformation from a body frame of reference to a user's frame ofreference.

During operation, the device estimates Q in an implementation dependentmanner to perform this transformation. One exemplary implementationdescribed above involves compensating for tilt (i.e., variations inx-axis roll of the 3D pointing device based on the manner in which it isheld by a user). The orientation is computed by first estimating theacceleration component due to gravity in the body frame, Ab. Bydefinition, the acceleration vector due to gravity in the user frame,Ag, is set to [0, 0,−1]. Since gravity cannot estimate the heading(rotation about the z-axis), the body frame estimate for heading isused. Therefore, the rotation quaternion has an axis of rotation in thez=0 plane. The following is one of several mathematically equivalentmethods for computing the rotation quaternion:V=∥Ab∥×∥Ag∥ (cross product of unit vectors)qV=∥V∥α=sin⁻¹ |V|Quaternion[qV,α]=[qV*sin(α/2), cos(α/2)]Position is then computed as the double integral of the acceleration inthe user frame. The acceleration in the user frame is the accelerationof the body frame rotated into the user frame by Q above. Normally, theorigin is assumed to be zero when the device is first activated, but theorigin may be reset during normal operation either manually orautomatically.

Generally, when the device is not moving, Pu′, Pu″, Wu, and Wu″ are all0. In this exemplary embodiment, Pb″ and Wb are measured. Although aninfinite number of rotations Q exist, the minimal rotation can beselected from the available set and used to estimate Wu based on Wb.Alternatively, Q may be computed using an assumed starting offsetorientation Qo, by integrating Wb over time as shown using the discretetime integral below:WbAngle=|Wb|*periodQ _(DELTA)=Quaternion[Wb, WbAngle]=[∥Wb∥*sin(WbAngle/2),cos(WbAngle/2)]Q _(NEXT) =Q ₀ **Q _(DELTA)Where * represents multiplication and ** represents quaternionmultiplication. Additional stability can be provided by constant fieldvectors including gravity and the earth's magnetic field and combinedwith the results above. The combination can be achieved using severalnumerical and filtering methods including, but not limited to, Kalmanfiltering.

A variety of different sensors could be employed as long as they measuremotion with respect to the body of the device. Exemplary sensors includeaccelerometers, rotational sensors, gyroscopes, magnetometers andcameras. The user frame does not need to be stationary. For example, ifthe user's frame of reference is selected to be the user's forearm, thenthe device would only respond to wrist and finger movement.

One skilled in the art will recognize the commutative property appliesto the frame of reference transformations described in this invention.Therefore, the order of the mathematical operations can be alteredwithout materially affecting the invention described herein. Inaddition, many motion processing algorithms can operate in either frameof reference equivalently, especially when the user frame is chosen tobe stationary with a constant orientation.

In addition to providing ease of use, frame of reference transformationsaccording to this exemplary embodiment of the present invention can alsobe used to address other challenges in handheld device implementations.For example, if a sensor (such as an accelerometer) is not locatedprecisely at the center of rotation in the body frame of reference, themeasured acceleration will include both the acceleration of the frameand acceleration components due to the rotation of the frame. Therefore,the measured acceleration can first be transformed to a different targetlocation within the body frame of the device using the followingrelationship:Abody=Aaccelerometer+ω′×R+ω×(ω×R)where R is the vector from the accelerometer to the target location, ωis the angular velocity of the body frame of reference and ω′ is theangular acceleration of the body frame of reference. If the body frameof the device is constructed such that it lies at R from theaccelerometer, then it should have zero angular acceleration effects andmay be more easily used to compute the device movement in the userframe. This compensates for intentional or unintentional misalignmentbetween the accelerometer and the center of the body frame of reference.In addition, the estimate of the gravity vector becomes much simplersince there are fewer forces acting at the center of rotation. Then,Auser=Rotate(Abody,Q)where Q is the rotation from the body frame of reference to theaccelerometer frame of reference.

Unfortunately, different users have different values for R. For example,one user may use the handheld device by rotating their elbow whileanother may use the device by rotating their wrist. In addition, peoplehave different sized wrists and forearms. For improved usability thisexemplary embodiment of the handheld dynamically computes R and movesthe body origin such that it has minimal acceleration components due toangular motion. The exemplary embodiment estimates R by defining R as[Rx, 0, 0] and solving for Rx using and minimizing Abody−Rotate[Ag, Q].Note that many numerical methods exist including recursive least squaresand Kalman filtering that may perform minimization to compute Rx.

Based on the foregoing, it will be appreciated that the presentinvention describes various techniques for mapping sensed motion of ahandheld device from one frame of reference (e.g., a body frame ofreference) to another frame of reference (e.g., a user's frame ofreference). These mappings can be independent from other mappingsassociated with the use of the handheld device, e.g., the mapping ofsensed motion to cursor movement or can be combined therewith. Moreover,transformations according to the present invention can be performed totransform the sensed motion in all three dimensions, for translationalmotion and rotational motion or any subset thereof, from the perspectiveof either the input side of the motion equation or the output side.Additionally, the selection of the frame of reference into which thesensed motion is mapped or transformed can be made in a number ofdifferent ways. One example provided above shows the second frame ofreference being a user's frame of reference associated with the tilt ofthe device, however many other variations are possible. For example, theuser may select his or her desired frame of reference, which setting canbe stored in the handheld as one of a plurality of user preferences andused to perform the transformation. The second frame of reference can beselected based on any number of techniques. The second frame ofreference can be selected based upon an explicit command (e.g., buttonor user interface selection) or automatically through user recognitiondetermined by device use patterns, tremor, and other biometrics.

Additionally, although some of the exemplary embodiments describe aboveoperate on data in the velocity domain, the present invention is not solimited. Mapping or transformation according to the present inventioncan alternatively or additionally be performed on, for example, positionor acceleration data and can be for translational motion, rotationalmotion or both. Also the order of processing is not critical. Forexample, if the handheld device is being used to output gesturecommands, the mapping can be performed first and then the gesturedetermined or the gesture can be determined first and then the mappingcan be performed.

The above-described exemplary embodiments are intended to beillustrative in all respects, rather than restrictive, of the presentinvention. Thus the present invention is capable of many variations indetailed implementation that can be derived from the descriptioncontained herein by a person skilled in the art. For example, althoughthe foregoing exemplary embodiments describe, among other things, theuse of inertial sensors to detect movement of a device, other types ofsensors (e.g., ultrasound, magnetic or optical) can be used instead of,or in addition to, inertial sensors in conjunction with theafore-described signal processing. All such variations and modificationsare considered to be within the scope and spirit of the presentinvention as defined by the following claims. No element, act, orinstruction used in the description of the present application should beconstrued as critical or essential to the invention unless explicitlydescribed as such. Also, as used herein, the article “a” is intended toinclude one or more items.

1. A method for using a free space pointing device comprising the stepsof: detecting movement of said free space pointing device using anaccelerometer and at least one other sensor; determining an orientation,in which said free space pointing device is held, based on an output ofsaid accelerometer; compensating said at least one other sensor'sdetected movement based on said determined orientation by performing atwo-dimensional rotational transform on said at least one other sensor'sdetected movement to generate an output which is substantiallyindependent of a tilt of said free space pointing device with referenceto a predetermined axis; and determining a gesture associated with thedetected movement.
 2. A handheld device comprising: a sensor configuredto generate a first output associated with motion of said handhelddevice; an accelerometer configured to detect acceleration of saidhandheld device and outputting at least one second output; and aprocessing unit configured to receive and process said first output fromsaid sensor and said at least one second output from said accelerometer,said process including: determining an orientation in which saidhandheld device is held using said at least one second output,compensating said first output based on said determined orientation byperforming a two-dimensional rotational transform on said first outputto generate an output which is substantially independent of a tilt ofsaid handheld device with reference to a predetermined axis; anddetermining a gesture associated with the detected movement.
 3. Thehandheld device of claim 2, wherein said sensor is a camera.
 4. Thehandheld device of claim 2, wherein said sensor is a rotational sensor.5. The handheld device of claim 2, wherein said sensor is amagnetometer.
 6. The handheld device of claim 2, wherein said sensor isan optical sensor.
 7. The handheld device of claim 2, wherein saidaccelerometer is a multi-axis accelerometer and wherein said at leastone second output includes a value y generated by said multi-axisaccelerometer associated with acceleration of said handheld device in ay-axis direction and a value z generated by said multi-axisaccelerometer associated with acceleration of said handheld device in az-axis direction.
 8. The handheld device of claim 7, wherein saidmulti-axis accelerometer is a 3-axis accelerometer.
 9. The handhelddevice of claim 2, wherein said first output is generated at a samplingrate of 200 samples/second.
 10. The handheld device of claim 2, furthercomprising: a wireless transceiver for receiving data from saidprocessing unit and wirelessly transmitting said data.
 11. The handhelddevice of claim 10, wherein said wireless transceiver is a Bluetooth®transceiver.
 12. The handheld device of claim 2, further comprising: aplurality of buttons disposed on a housing of said handheld device; andat least one LED disposed on said housing.
 13. The handheld device ofclaim 2, wherein said first output includes position data values. 14.The handheld device of claim 2, wherein said processing performs saidtwo dimensional rotational transform on detected translational motion ofsaid handheld device.
 15. The handheld device of claim 2, wherein saidprocessing performs said two dimensional rotational transform ondetected rotational motion of said handheld device.
 16. The handhelddevice of claim 2, wherein said processing performs said two dimensionalrotational transform on both detected translational motion and detectedrotational motion of said handheld device.
 17. The handheld device ofclaim 2, wherein said handheld device is a 3D pointing device.
 18. Asystem comprising: (a) a handheld device including: a sensor configuredto generate a first output associated with motion of said handhelddevice; and an accelerometer configured to detect acceleration of saidhandheld device and outputting at least one second output; and (b) aprocessing unit for receiving and processing said first output from saidsensor and said at least one second output from said accelerometer, saidprocessing including: determining an orientation in which said handhelddevice is held using said at least one second output; compensating saidfirst output based on said determined orientation by performing atwo-dimensional rotational transform on said first output to generate anoutput which is substantially independent of a tilt of said handhelddevice with reference to a predetermined axis; and determining a gestureassociated with the motion of the handheld device.
 19. The system ofclaim 18, wherein said system further comprises: a system controller; awireless transceiver, disposed in said handheld device, for transmittingdata from said handheld device to said system controller; wherein saidprocessing unit is disposed within one of said handheld device and saidsystem controller.
 20. The system of claim 18, wherein said sensor is acamera.
 21. The system of claim 18, wherein said sensor is a rotationalsensor.
 22. The system of claim 18, wherein said sensor is amagnetometer.
 23. The system of claim 18, wherein said sensor is anoptical sensor.
 24. The system of claim 18, wherein said accelerometeris a multi-axis accelerometer and wherein said at least one secondoutput includes a value y generated by said multi-axis accelerometerassociated with acceleration of said handheld device in a y-axisdirection and a value z generated by said multi-axis accelerometerassociated with acceleration of said handheld device in a z-axisdirection.
 25. The system of claim 24, wherein said multi-axisaccelerometer is a 3-axis accelerometer.
 26. The system of claim 18,wherein said first output is generated at a sampling rate of 200samples/second.
 27. The system of claim 19, wherein said wirelesstransceiver is a Bluetooth® transceiver.
 28. The system of claim 18,further comprising: a plurality of buttons disposed on a housing of saidhandheld device; and at least one LED disposed on said housing.
 29. Thesystem of claim 18, wherein said first output includes position datavalues.
 30. The system of claim 18, wherein said processing unitperforms said two-dimensional rotational transformation on translationalmotion detected by said sensor.
 31. The system of claim 18, wherein saidprocessing unit performs said two-dimensional rotational transform onrotational motion detected by said sensor.
 32. The system of claim 18,wherein said processing unit performs said two-dimensional rotationaltransform on both translational motion and rotational motion detected bysaid sensor.
 33. The system of claim 18, wherein at least one of saidfirst output and said at least one second output are processed prior tosaid compensating.
 34. The system of claim 33, wherein said processingprior to said compensating includes compensating said at least one ofsaid first output and said at least one second output for offset bias.35. The system of claim 33, wherein said processing prior tocompensating includes converting said at least one of said first outputand said at least one second output into different units.
 36. The deviceof claim 2, wherein at least one of said first output and said at leastone second output are processed prior to said compensating.
 37. Thedevice of claim 36, wherein said processing prior to said compensatingincludes compensating said at least one of said first output and said atleast one second output for offset bias.
 38. The device of claim 36,wherein said processing prior to said compensating includes convertingsaid at least one of said first output and said at least one secondoutput into different units.
 39. The system of claim 18, wherein saidhandheld device is a 3D pointing device.
 40. A method comprising:generating, from a first sensor, a first output associated with motionof a handheld device; detecting, by a second sensor, acceleration ofsaid handheld device and outputting at least one second output; andprocessing said first output and said at least one second output, saidprocessing including: determining an orientation in which said handhelddevice is held using said at least one second output; compensating saidfirst output based on said determined orientation by performing atwo-dimensional rotational transform on said first output to generate anoutput which is substantially independent of a tilt of said handhelddevice with reference to a predetermined axis; and determining a gestureassociated with the motion of the handheld device.
 41. The method ofclaim 40, further comprising: transmitting data from said handhelddevice to a system controller, wherein said processing is performedwithin one of said handheld device and said system controller.
 42. Themethod of claim 40, wherein said first sensor is a camera.
 43. Themethod of claim 40, wherein said first sensor is a rotational sensor.44. The method of claim 40, wherein said first sensor is a magnetometer.45. The method of claim 40, wherein said first sensor is an opticalsensor.
 46. The method of claim 40, wherein said second sensor is amulti-axis accelerometer and wherein said at least one second outputincludes a value y generated by said multi-axis accelerometer associatedwith acceleration of said handheld device in a y-axis direction and avalue z generated by said multi-axis accelerometer associated withacceleration of said handheld device in a z-axis direction.
 47. Themethod of claim 46, wherein said multi-axis accelerometer is a 3-axisaccelerometer.
 48. The method of claim 40, wherein said first output isgenerated at a sampling rate of 200 samples/second.
 49. The method ofclaim 41, wherein said step of transmitting further comprises:wirelessly transmitting said data in accordance with Bluetooth®.
 50. Themethod of claim 40, further comprising: disposing a plurality of buttonsand at least one LED on a housing of said handheld device.
 51. Themethod of claim 40, wherein said first output includes position datavalues.
 52. The method of claim 40, wherein said processing performssaid two-dimensional rotational transform on translational motiondetected by said first sensor.
 53. The method of claim 40, wherein saidprocessing performs said two-dimensional rotational transform onrotational motion detected by said first sensor.
 54. The method of claim40, wherein said processing performs said two-dimensional rotationaltransform on both translational motion and rotational motion detected bysaid first sensor.
 55. The method of claim 40, further comprising:processing at least one of said first output and said at least onesecond output prior to said compensating.
 56. The method of claim 55,wherein said processing prior to said compensating includes compensatingsaid at least one of said first output and said at least one secondoutput for offset bias.
 57. The method of claim 55, wherein saidprocessing prior to said compensating includes converting said at leastone of said first output and said at least one second output intodifferent units.
 58. The method of claim 40, wherein said handhelddevice is a 3D pointing device.
 59. A system comprising: means forgenerating, from a first sensor, a first output associated with motionof a handheld device; means for detecting, by a second sensor,acceleration of said handheld device and outputting at least one secondoutput; and means for processing said first output and said at least onesecond output, said processing means including: means for determining anorientation in which said handheld device is held using said at leastone second output; means for compensating said first output based onsaid determined orientation by performing a two-dimensional rotationaltransform on said first output to generate an output which issubstantially independent of a tilt of said handheld device withreference to a predetermined axis; and means for determining a gestureassociated with the motion of the handheld device.
 60. The system ofclaim 59, wherein said system further comprises: means for controlling;and means for transmitting data, disposed in said handheld device, fromsaid handheld device to said means for controlling, wherein said meansfor processing is disposed within one of said handheld device and saidmeans for controlling.
 61. The system of claim 59, wherein said firstsensor is a camera.
 62. The system of claim 59, wherein said firstsensor is a rotational sensor.
 63. The system of claim 59, wherein saidfirst sensor is a magnetometer.
 64. The system of claim 59, wherein saidfirst sensor is an optical sensor.
 65. The system of claim 59, whereinsaid second sensor is a multi-axis accelerometer and wherein said atleast one second output includes a value y generated by said multi-axisaccelerometer associated with acceleration of said handheld device in ay-axis direction and a value z generated by said multi-axisaccelerometer associated with acceleration of said handheld device in az-axis direction.
 66. The system of claim 65, wherein said multi-axisaccelerometer is a 3-axis accelerometer.
 67. The system of claim 59,wherein said first output is generated at a sampling rate of 200samples/second.
 68. The system of claim 59, wherein said means fortransmitting data is a Bluetooth® transceiver.
 69. The system of claim59, further comprising: a plurality of buttons disposed on a housing ofsaid handheld device; and at least one LED disposed on said housing. 70.The system of claim 59, wherein said first output includes position datavalues.
 71. The system of claim 59, wherein said means for processingperforms said two-dimensional rotational transformation on translationalmotion detected by said first sensor.
 72. The system of claim 59,wherein said means for processing performs said two-dimensionalrotational transform on rotational motion detected by said first sensor.73. The system of claim 59, wherein said means for processing performssaid two-dimensional rotational transform on both translational motionand rotational motion detected by said first sensor.
 74. The system ofclaim 59, wherein at least one of said first output and said at leastone second output are processed prior to said compensating.
 75. Thesystem of claim 74, wherein said processing prior to said compensatingincludes compensating said at least one of said first output and said atleast one second output for offset bias.
 76. The system of claim 74,wherein said processing prior to compensating includes converting saidat least one of said first output and said at least one second outputinto different units.
 77. The system of claim 59, wherein said handhelddevice is a 3D pointing device.
 78. The method of claim 1, furthercomprising: transmitting data from said handheld device to a systemcontroller, wherein said determining and compensating are performedwithin one of said handheld device and said system controller.
 79. Themethod of claim 1, wherein said at least one other sensor is a camera.80. The method of claim 1, wherein said at least one other sensor is arotational sensor.
 81. The method of claim 1, wherein said at least oneother sensor is a magnetometer.
 82. The method of claim 1, wherein saidat least one other sensor is an optical sensor.
 83. The method of claim1, wherein said accelerometer is a multi-axis accelerometer and whereinan output of said accelerometer includes a value y generated by saidmulti-axis accelerometer associated with acceleration of said handhelddevice in a y-axis direction and a value z generated by said multi-axisaccelerometer associated with acceleration of said handheld device in az-axis direction.
 84. The method of claim 83, wherein said multi-axisaccelerometer is a 3-axis accelerometer.
 85. The method of claim 1,wherein an output of said at least one other sensor is generated at asampling rate of 200 samples/second.
 86. The method of claim 78, whereinsaid step of transmitting further comprises: wirelessly transmittingsaid data in accordance with Bluetooth®.
 87. The method of claim 1,further comprising: disposing a plurality of buttons and at least oneLED on a housing of said handheld device.
 88. The method of claim 1,wherein an output of said at least one other sensor includes positiondata values.
 89. The method of claim 1, wherein said compensatingperforms said two-dimensional rotational transform on translationalmotion detected by said at least one other sensor.
 90. The method ofclaim 1, wherein said compensating performs said two-dimensionalrotational transform on rotational motion detected by said at least oneother sensor.
 91. The method of claim 1, wherein said compensatingperforms said two-dimensional rotational transform on both translationalmotion and rotational motion detected by said at least one other sensor.92. The method of claim 1, further comprising: processing at least oneof an output of said accelerometer and an output of said at least oneother prior to said compensating.
 93. The method of claim 92, whereinsaid processing prior to said compensating includes compensating said atleast one of said output of said accelerometer and said output of saidat least one other sensor for offset bias.
 94. The method of claim 92,wherein said processing prior to said compensating includes convertingsaid at least one of said output of said accelerometer and said outputof said at least one other sensor into different units.
 95. The methodof claim 2, wherein said handheld device is a 3D pointing device.